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ABSTRACT 



Instructional metacognition refers both to conceptions of 
students about the contribution that instructional interventions can make to 
their learning, and to the impact of these conceptions on students' 
interpretation and use of instructional interventions. This study analyzes 
student conceptions of efficiency-related attributes of instructional 
interventions- -instructional elements or features of them, such as lecture, 
courseware, transparencies during a lecture, cartoons in a textbook; and 
learning activities (activities both initiated and executed by the student) 
such as making a summary or discussions with peers. A survey study of 
university freshman (n=489) was re-analyzed to address these issues. The 
questionnaire contained identification questions, and a list of 20 
instructional interventions and 20 learning activities to be rated on a 
5-point Likert-type scale for their contribution to study result and study 
time. Gender, domain of study, and educational background were analyzed as 
independent variables. Results include: (1) university freshman largely 

perceive current school practices of both interventions and learning 
activities to be highly efficient; (2) students perceive that interventions 
that contribute to an increase of study results decrease study time, and 
learning activities that increase study time also increase study results; (3) 
for both instructional interventions and learning activities, students reject 
the use of technology as being inefficient; and (4) domain of study is the 
only independent variable that consistently affects students' conceptions of 
instructional interventions. (Contains 13 references.) (SWC) 
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Introduction 



Research in educational psychology and instructional design has demonstrated that effects of 
instructional interventions are mediated by the interpretation and use by the learner (e.g. Elen, 1995; 
Winne & Marx, 1982). In order to explain these findings, it has been suggested (Elen & Lowyck, 

1996; Lowyck & Elen, 1994) that in addition to learning-related metacognition which directs learning 
activities, another type of metacognition, instructional metacognition, must also be considered. 
Instructional metacognition may direct the interpretation and use of instructional interventions. It refers 
first of all to conceptions of students about the contribution that instructional interventions can make to 
their learning. As for metacognition in general (Brown, 1978; Flavell, 1979), it also refers to the 
impact of these conceptions on the interpretation and use of instructional interventions by students 
(Figure 1). 




Figure 1 : Instructional metacognition as a variable mediating the effects of instructional interventions 

Whereas the concept of instructional metacognition may be important as it provides a possibly 
powerful explanation of the lack of direct effects of regular instructional environments, it becomes 
even more important for so-called ‘open’ learning environments, or instructional settings in which the 
control is transferred from the environment or programme to the learner. In such settings, the learner 
him- or herself decides about the use of environmental elements in the absence of specific instructional 
incentives. An adequate selection and use of environmental elements for learning purposes requires an 
in-depth understanding of the potential of these elements for one’s learning. In other words, in more 
open learning environments, learners become more responsible for their own learning and hence have 
to turn these environments themselves into instructionally valuable ones. They have to become their 
own instructional designers. Students’ knowledge of instructional design, or broader, their conceptions 
about the relationship between environmental elements and their own learning, will determine the use 
they make of those environmental elements and hence about the effectiveness of those elements and 
their effects on learning. 
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In the recent past, some studies have addressed aspects of instructional metacognition. Some 
researchers, for instance, have investigated questions related to the content and structure of 
instructional metacognition and discrepancies between conceptions of students and of teachers (e.g. 
Levy, Wubbels, & Brekelmans, 1996; Stebler & Reusser, 1996). Other researchers, such as Ertmer and 
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Newby (1996) explored the impact of conceptions about case-based instruction on learning activities. 
Notwithstanding these studies, research on issues related to instructional metacognition is limited. 

The complexity of instructional metacognition calls for a systematic research program (see : Lowyck & 
Elen, 1994). Given the variety of instructional interventions and the complexity of their relationship 
with learning, research on instructional metacognition within such a program can follow two main 
methodological directions. Studies may focus on one or a limited number of instructional interventions 
or aspects of the learning environment and investigate how different attributes of these interventions 
relevant to learning and instruction are conceived by students and affect learning (for case-based 
instruction, e.g. : Ertmer & Newby, 1996; for peer collaboration, e.g. : Stebler & Reusser, 1996). Or in 
a second approach, studies may focus on one or a limited number of attributes (e.g. effectiveness, 
structuredness). In this alternative approach a large number of instructional interventions or aspects of 
the learning environment may get investigated for a limited set of attributes. 

In our own research, the second approach has been followed in a study analysing conceptions of 
students about efficiency-related attributes of instructional interventions and learning (Elen & Lowyck, 
1996). Because data from this study constitute the starting point for the analysis presented in this paper, 
a brief summary of the findings seems appropriate. An analysis of survey data regarding efficiency- 
related attributes of instructional interventions and learning activities has revealed that (1) university 
freshmen are very obedient as they largely accept current school practices as being efficient. They 
regard both interventions (e.g. lectures) and learning activities (e.g. repeat regularly) mainly directed 
toward surface-level processing and reproduction to be highly efficient. These opinions of students 
reflect research on effective teaching in which the following characteristics of effective teaching have 
been revealed : clarity, structure, opportunity to learn criterion material, focus on academic activities 
(Dunkin & Biddle, 1974). (2) Whereas in students’ conceptions there is a negative correlation between 
contributions to study results and study time of instructional interventions except for exams, a positive 
correlation is found between these two types of contributions for learning activities. This means that in 
students’ conceptions int&iVciuitm* 'mat contribute to an increase of study results, decrease stuuy 
whereas learning activities that increase study time also increase study results (for an overview see 
Figure 2 and Figure 3). (3) For both instructional interventions and learning activities, students reject 
the use of technology as being inefficient. As Kinzie, Delcourt, and Powers (1994) suggest, this may 
be closely related to low feelings of self-efficacy with computers and limited experience with these 
instructional media. It may also be a reversed novelty-effect. (4) Finally, while some differences have 
been found, only domain of study (which of course is related to the specific learning environment 
students are experiencing) seemed to consistently affect students’ conceptions of instructional 
interventions. 
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Figure 2 : The relationship between contribution to study time and study result for instructional 
interventions 
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Figure 3 : The relationship between contribution to study time and study result for learning activities 

Research questions 

The study mentioned above suggested large homogeneity in students’ conceptions about efficiency- 
related attributes of instructional interventions. In view of the elaboration of an instrument to more 
precisely assess ‘instructional metacognition’, the issue of homogeneity of students’ conceptions was 
given priority in our research. The implications are indeed important. With a homogeneous set of 
conceptions, students can be treated as one group when these conceptions need change. At the same 
time, however, absolute homogeneity would provide no insight as to the origin and development of 
instructional metacognition. Hence, in case of homogeneity, there is no indication on where precisely 
to start to change conceptions. 
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At the core of this study, then, is the question whether different groups of students can be identified 
with respect to their conceptions about efficiency-related attributes of instructional interventions and 
learning activities. More concretely, it has been investigated whether groups of students can be 
identified that differ from one another in their rating of two attributes of instructional interventions and 
learning activities, namely contribution to study time and contribution to study result. Both 
instructional interventions and learning activities were considered as to further investigate the 
uniqueness of the concept of instructional metacognition. 

Method 

In order to answer the above mentioned questions data from a survey study that investigated the 
conceptions of students about the efficiency of instructional interventions and learning activities were 
re-analysed (Elen & Lowyck, 1996). Instructional interventions refer to instructional elements (e.g. 
lecture, courseware) or features of such elements (e.g. transparancies during lectures, cartoons in a 
coursetext). Learning activities refer to activities that are both initiated and executed by the student 
(e.g. making a summary, discussing with peers). In the survey study, university freshmen (N=489) 
ratings of efficiency-related attributes of instructional interventions and learning activities were 
investigated. The questionnaire consisted of (1) a number of identification questions (gender, domain 
of study and educational background), (2) a list of 20 instructional interventions and (3) a list of 20 
learning activities both to be rated on a five point Likert-type scale for their contribution to study result 
and study time. 

In order to get a more profound understanding of the origins of differences in students’ conceptions 
gender, domain of study and educational background were included in the analysis as independent 
variables. 

Repeatedly, past research has shown female students to have a more negative attitude toward 
computers than male students (for a review Lowyck, Elen, Proost, & Buena, 1995). It remains to be 
studied, though, whether gender differences in perceptions are typical for technological means or are 
more generic and do also appear when regular instructional interventions are investigated. Besides 
gender, it might be expected that given different recruitment profiles on the one hand and different 
instructional approaches on the other, students in different domains of study differ in their conceptions 
about the efficiency of both instructional interventions and learning activities. Therefore, in addition to 
the general question with respect to instructional metacognitive knowledge, gender and study domain 
differences are also included. Finally, in view of exploring the development of instructional 
metacognition, one may wonder whether differences in educational background also result in different 
perceptions about instructional interventions. With respect to educational background, level of 
secondary education has been considered. 
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Students from two domains of study have participated : educational sciences and psychology (see 
Table 1). As it is the case in the population of these two domains of study, female students clearly 
outnumber male students in the sample. 

Table 1 : Description of sample : gender and domain of study 



N 


educational 

sciences 


psychology 


total 


male 


14 


64 


78 

15.95 % 


female 


192 


219 


411 

84.05 % 


total 


211 

42.13 % 


288 

57.87 % 


489 

100.00 % 



A second variable refers to an attribute of educational background, i.e. difficulty level of secondary 
education (Table 2). In view of determining the level of their secondary education, students were 
requested to indicate the number of hours of mathematics, Latin and Greek they attended weekly 
during the last year of secondary education. In Flanders the more hours followed in these domains the 
more difficult studies are regarded to be. Three groups of students can be identified : a high level group 
for students with 10 or more hours in these domains, a middle level group with less than 10 but more 
than 5 and a low level group with 5 or less than 5 hours of mathematics, Greek and/or Latin. Both 
generation (freshmen) and non-generation students (students who double their first year of study at the 
university) participated. 

Table 2 : Description of sample : educational background 



N 


generation 


non- 

generation 


total 


high level 


148 


15 


163 

33 . 33 % 


middle level 


180 


20 


200 

40 . 90 % 


low level 


103 


23 


126 

25 . 77 % 


total 


431 

88 . 14 % 


58 

11 . 86 % 


489 

100 . 00 % 



Questionnaires were administered during regular lecturing times after six months of experience with 
university teaching but before generation students experienced a cram-period. Questionnaires were 
filled out during regular lectures separately for psychology and educational science students. In total 
489 students participated. However, complete data were available for 422 students. 
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Students rated 20 instructional elements and 20 learning activities on their contribution to overall study 
time and to learning outcome separately. Because the goal was to tap general conceptions of university 
freshmen, no specific context was provided. Students were instructed to think about instructional 
environments familiar to them and then rate the different elements and activities. 

Results 

In view of identifying groups of students, data were first analysed using principal component analysis 
with orthogonal varimax rotation as specified in the statistical package STATISTICA. Based on an 
analysis of the scree-plot and the eigenvalue criterion of one, a solution with three factors was selected. 
This solution explains 26.01% of the total variance. 

Factor 1 explains 11.1 5% of the total variance. All items with high loadings (>= .40) on factor 1 
address ratings by students of contributions to study time of instructional interventions and learning 
activities. Highest loadings (>= .75) are found for the ratings on contributions to study time of three 
learning activities : ‘attend practical sessions’, ‘repeat regularly’, and ‘attend lectures’. Instructional 
interventions with highest loadings (>= .50) on this factor are : lectures, tutorial sessions, course notes 
written by a university teacher, and structure on the blackboard. Given these results the first factor may 
be said to refer to contribution to study time, with highest loadings for more traditional instructional 
interventions and learning activities immediately related to well-known instructional interventions that 
are externally steering the learning activities and processes. 

Factor 2 explains 9.34% of the total variance. All items with high loadings (>= .40) on factor 2 address 
ratings by students of contributions of learning activities to study results. Two observations with 
respect to factor loadings on factor 2 are to be mentioned. First, all ratings of contributions of learning 
activities to study results load on factor 2 except for ‘looking for information on Internet’, ‘exercise 
with some piece of courseware’ and ‘go to the library’. Second, none of the ratings of contributions of 
instructional interventions to study results load have factor loadings higher than .40 on factor 2. 

Highest loading on this factor (>=.70) are for ‘attend practical sessions’, ‘repeat regularly’, ‘attend 
lectures’ and ‘invent examples oneself. It can be concluded therefore that factor 2 refers to an 
assessment of conceptions about the effectiveness of well-known learning activities of university 
students. Highest loadings are found for activities that are appropriate in a direct teaching environment. 

Factor 3 explains 5.52% of the total variance. All items with high loadings (>= .40) on factor 3 address 
ratings by students of contributions to study results of instructional interventions. Highest loadings 
(>=.55) are for the following instructional interventions : ‘lectures’, ‘a handbook’, ‘tutorial sessions’ 
and ‘examples of exam questions’. Factor 3 then may be interpreted as referring to the contribution of 
traditional instructional interventions to study results. 
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As a first step towards the identification of groups of students, factor scores of individual subjects were 
plotted for each combination of factors (Figure 3). It can be seen in these scatter plots that conceptions 
of students do vary most for factor 1 . Factor scores are calculated as the standardized sum of subjects 
ratings for all items weighted by factor loadings. 




Figure 3 : Plots for combinations of factors 

Next a cluster analysis using K-means clustering has been executed. In this approach the number of 
clusters is determined a priori. In the analysis which can be conceived as an inversed ANOVA, clusters 
are constructed in order to minimize variability within clusters and maximize variability between 
clusters. F values provide an indication of the significance of the differences between clusters. In this 
study, analyses with 2 or more clusters revealed the same overall results. The most parsimonous 
solution with two clusters will be presented. 

A first cluster analysis on factorscores with the three factors as variables reveals two clusters. 




Figure 4 : Plot of means for two clusters 
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Clusters differ from one another significantly for factor 1 (F(l; 382) = 869.035, p<.000) but not for 
factor 2 and only marginally for factor 3 (F(l;382) = 4.280, p < .05). In other words, two groups of 
students can be detected that differ from one another based on their ratings of contributions to study 
time of traditional instructional interventions and learning activities immediately related to these well- 
known instructional interventions (Figure 4). 

In order to further describe the two groups, a new cluster analysis was made considering factor scores 
on the three factors together with the above mentioned independent variables : gender, domain of study 
and educational background. Educational background here refers to the estimated difficulty level of 
secondary education of respondents. This selection of independent variables is exploratory. At this 
stage, it is unknown which variables mostly affect aspects of instructional metacognition. 

Again a solution with two clusters was the most parsimonous one. As in the first analysis, these 
clusters differ from one another with respect to factor 1 (F(l; 382)=1 1.533; p< 00) but also with 
respect to educational background (F(l; 382)=1074.389; p<00). No differences between clusters are 
found anymore for factor 3. In other words (see Table 3, Figure 5), students who estimate that 
traditional instructional interventions and learning activities immediately related to these well-known 
instructional interventions reduce study time, have a ‘stronger’ educational background whereas 
students with a ‘weaker’ educational background think these interventions and learning activities rather 
to contribute to an increase of study time. It is important to note that both groups do not differ with 
respect to the estimated contribution to study results of instructional interventions and learning 
activities. 

Table 3 : Means for clusters 





Cluster 1 


Cluster 2 


Gender 


1.17 


1.18 


Domain 


1.44 


1.38 


Level 


4.77 


10.00 


Factor 1 : contribution to study time of learning 


-.17 


.17 


activities and instructional interventions* 






Factor 2 : contibution to study results of learning 


.01 


-.01 


activities** 






Factor 3 : contribution to study results of instructional 


.00 


-.00 


interventions** 







* a lower score indicates a contribution that increases study time, a higher score one 
that decreases study time 



** a lower score indicates a contribution that increases study result, a higher score one 
that decreases study result 
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Cluster 
No. 1 




Cluster 
No. 2 



Figure 5 : Plot of means for two clusters with three factors and three independent variables 

Conclusions 

In this paper data from a survey study have been re-analysed in order to explore further the 
homogeneity of students’ conceptions. This study is needed in view of the elaboration of an adequate 
instrument to measure instructional metacognition. It has been found that students conceive 
contributions of both instructional interventions and learning activities in a similar way. This suggests 
to make no difference between learning-related and instructional metacognition. However, for 
contribution to study results, results of principal component analysis reveals two factors, one for 
learning activities and another for instructional interventions. This, then, suggests to distinguish 
instructional metacognition from learning-related metacognition. 

Cluster analyses have revealed two clusters. These clusters do not differ with respect to factor scores 
relating to contributions to study results, gender and domain of study. Clusters differ with respect to 
conceptions about contributions to study time of both learning activities and instructional interventions 
and level of educational background. 

Students seem to differentially appreciate effectiveness and efficiency criteria. Effectiveness criteria 
are commonly perceived as important by all types of students, while students’ conceptions differ with 
respect to efficiency criteria. Consequently, this research on a mediating variable, instructional 
metacognition, suggests to take into account these different positions and design different learning 
environments for reaching efficient instruction and learning. 

The outcome of the cluster analysis is difficult to interprete. Several explanations could be brought 
forward. Results may suggest that students with a stronger educational background have experienced 
more frequently the contribution of traditional interventions and associated learning activities to a 
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reduction of total study time. This may induce them to attend actively to these traditional interventions 
and engage in the specified learning activities. This explanation may also imply that students with a 
stronger educational background have been and will be more successful in their study. They are more 
eager to engage in adequate learning activities and use instructional interventions as they assume this 
to reduce study time. This implication is to be further studied. Similarly, the higher success rate is also 
to be investigated. Another explanation may be that students with a stronger educational background 
do consider contribution to study time as an issue whereas students with a weaker background do not. 
Students with a stronger educational background may have had to consider time when confronted with 
high demands in secondary education, whereas students with a weaker background may not have 
experienced any urge to actually consider time as an issue during studying. They may not see the 
relationship between study time and study result. 

It needs further study to better understand why higher and lower level students conceive contributions 
to study time differently but not contributions to study result. It could be that a confrontation with 
questions that refer to a usual, regular situation does not elicit differential answers, since it lacks profile 
and thus hinders well-balanced and intruding reflection processes. 

In terms of Derry (1996), the results suggest that with respect to study results, the mental objects and 
the cognitive field of students with both a strong and a weak educational background are similar. These 
similarities result in non-significantly different ratings of the contributions to study results of learning 
activities and instructional interventions. The different ratings for study time however reveal 
differences with respect to cognitive field and/or mental objects. This is to be further studied. The 
results also suggest that the relationship of learning activities and instructional interventions on the one 
hand and study result on the other is more evident than the relationship with study time. It might be 
that the estimated contribution to study results belongs to the same mental object as the learning 
activities and the instructional interventions, whereas contribution to study time only belongs to the 
cognitive field not the mental object of learning activities or instructional interventions. 

This contribution is to be regarded part of a broader endeavour in which we want to construct a valid 
instrument to measure instructional metacognition and to identify different groups of students. The 
availability of such an instrument would enable us to monitor evolutions in students’ instructional 

metacognition. It would also enable to design experiments with instructional metacognition as an 

\ 

important independent variable. The results add to the challenge to construct such an instrument. 
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Congratulations on being a presenter at AERA 1 . The ERIC Clearinghouse on Assessment and 
Evaluation invites you to contribute to the ERIC database by providing us with a printed copy of 
your presentation. 

Abstracts of papers accepted by ERIC appear in Resources in Education (RIE) and are announced 
to over 5,000 organizations. The inclusion of your work makes it readily available to other 
researchers, provides a permanent archive, and enhances the quality of RIE. Abstracts of your 
contribution will be accessible through the printed and electronic versions of RIE. The paper will 
be available through the microfiche collections that are housed at libraries around the world and 
through the ERIC Document Reproduction Service. 

We are gathering all the papers from the AERA Conference. We will route your paper to the 
appropriate clearinghouse. You will be notified if your paper meets ERIC's criteria for inclusion 
in RIE: contribution to education, timeliness, relevance, methodology, effectiveness of 
presentation, and reproduction quality. You can track our processing of your paper at 
http : //er icae2 . educ . cua. edu . 

Please sign the Reproduction Release Form on the back of this letter and include it with two copies 
of your paper. The Release Form gives ERIC permission to make and distribute copies of your 
paper. It does not preclude you from publishing your work. You can drop off the copies of your 
paper and Reproduction Release Form at the ERIC booth (523) or mail to our attention at the 
address below. Please feel free to copy the form for future or additional submissions. 

Mail to: AERA 1997/ERIC Acquisitions 



This year ERIC/AE is making a Searchable Conference Program available on the AERA web 
page (http://aera.net). Check it out! 



The Catholic University of America 
O’Boyle Hall, Room 210 
Washington, DC 20064 




Lawrence M. Rudner, Ph.D. 
Director, ERIC/AE 



'If you are an AERA chair or discussant, please save this form for future use. 



lEHICl Clearinghouse on Assessment and Evaluation 



